{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FS9T8jTqkZWI"
      },
      "source": [
        "# Use external tools with generate_text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f03e6e380116"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://ai.google.dev/examples/text_calculator\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on ai.google.dev</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/examples/text_calculator.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/google/generative-ai-docs/blob/main/site/en/examples/text_calculator.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D64fjiFDke_P"
      },
      "source": [
        "For some use cases, you may want to stop the generation from a model to insert specific results. For example, language models may have trouble with complicated arithmetic problems like word problems.\n",
        "This tutorial shows an example of using an external tool with the `genai.generate_text` method to output the correct answer to a word problem.\n",
        "\n",
        "This particular example uses the [`numexpr`](https://github.com/pydata/numexpr) tool to perform the arithmetic but you can use this same procedure to integrate other tools specific to your use case. The following is an outline of the steps:\n",
        "\n",
        "1. Determine a `start` and `end` tag to demarcate the text to send the tool.\n",
        "1. Create a prompt instructing the model how to use the tags in its result.\n",
        "1. Include the `end` tag in the of `stop_sequences` passed to `generate_text`.\n",
        "1. From the model result, take the text between the `start` and `end` tags as input to the tool.\n",
        "1. Run the tool and add its output to the prompt.\n",
        "1. Call `generate_text` again, to have the model continue with the tool's output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRbP4BER4_L9"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "oq3EYtJYBXpG"
      },
      "outputs": [],
      "source": [
        "!pip install -q google-generativeai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "h2rFkzdEKHtW"
      },
      "outputs": [],
      "source": [
        "import google.generativeai as genai\n",
        "genai.configure(api_key='YOUR API KEY')\n",
        "\n",
        "from google.api_core import retry\n",
        "\n",
        "@retry.Retry()\n",
        "def generate_text(*args, **kwargs):\n",
        "  return genai.generate_text(*args, **kwargs)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "PL3XpOnN7Yox"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "models/text-bison-001\n"
          ]
        }
      ],
      "source": [
        "models = [m for m in genai.list_models() if 'generateText' in m.supported_generation_methods]\n",
        "model = models[0].name\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zhZJMqsI7K61"
      },
      "source": [
        "## Try to solve the problem directly"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MhjnsEM67Hsh"
      },
      "source": [
        "Here's the word problem you're going to solve:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1TIekNZ1rrOd"
      },
      "outputs": [],
      "source": [
        "question = \"\"\"\n",
        "I have 77 houses, each with 31 cats.\n",
        "Each cat owns 14 mittens, and 6 hats.\n",
        "Each mitten was knit from 141m of yarn, each hat from 55m.\n",
        "How much yarn was needed to make all the items?\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OXRjf_hAEV1C"
      },
      "outputs": [],
      "source": [
        "prompt_template = \"\"\"\n",
        "You are an expert at solving word problems. Here's one:\n",
        "\n",
        "{question}\n",
        "\n",
        "Work through it step by step, and show your work.\n",
        "One step per line.\n",
        "\n",
        "Your solution:\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hTJIpg-D7w-3"
      },
      "source": [
        "Try it as is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IRoDEoU1Bq_6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "In the houses there are 77 * 31 = 2387 cats.\n",
            "So they need 2387 * 14 = 33418 mittens.\n",
            "And they need 2387 * 6 = 14322 hats.\n",
            "In total they need 33418 * 141 + 14322 * 55 = 5554525m of yarn.\n",
            "The answer: 5554525.\n"
          ]
        }
      ],
      "source": [
        "completion = generate_text(\n",
        "    model=model,\n",
        "    prompt=prompt_template.format(question=question),\n",
        "    # The maximum length of the response\n",
        "    max_output_tokens=800,\n",
        ")\n",
        "\n",
        "print(completion.result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zG4bukqCIQ4f"
      },
      "source": [
        "The prompt as is usually returns an incorrect result.\n",
        "It generally gets the steps right but the arithmetic wrong.\n",
        "\n",
        "The answer should be:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Jw0L9OoCAHZ"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5499648"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "answer = 77*31*14*141 + 77*31*6*55\n",
        "answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IAdE14kV7y0E"
      },
      "source": [
        "## Tell the model to use a calculator"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91LETezvs6LU"
      },
      "source": [
        "In this next attempt, give the model instructions on how to access the calculator. You can do that by specifying a `start` and `end` tag the model can use to indicate where a calculation is needed. Add something like the following to the prompt:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kjrZYUE7N5N6"
      },
      "outputs": [],
      "source": [
        "calc_prompt_template = \"\"\"\n",
        "You are an expert at solving word problems. Here's a question:\n",
        "\n",
        "{question}\n",
        "\n",
        "-------------------\n",
        "\n",
        "When solving this problem, use the calculator for any arithmetic.\n",
        "\n",
        "To use the calculator, put an expression between <calc></calc> tags.\n",
        "The answer will be printed after the </calc> tag.\n",
        "\n",
        "For example: 2 houses  * 8 cats/house = <calc>2 * 8</calc> = 16 cats\n",
        "\n",
        "-------------------\n",
        "\n",
        "Work through it step by step, and show your work.\n",
        "One step per line.\n",
        "\n",
        "Your solution:\n",
        "\"\"\"\n",
        "\n",
        "calc_prompt = calc_prompt_template.format(question=question)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2PhgztglJGv"
      },
      "source": [
        "To give the model access to the output of this \"calculator\", you'll have to pause generation and insert the result. Use the `stop_sequences` argument to stop at the `</calc>` tag:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7IPLT3yYk07Y"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "In each house, there are <calc>31 * 14\n"
          ]
        }
      ],
      "source": [
        "completion = generate_text(\n",
        "    model=model,\n",
        "    prompt=calc_prompt,\n",
        "    stop_sequences=[\"</calc>\"],\n",
        "    # The maximum length of the response\n",
        "    max_output_tokens=800,\n",
        "    candidate_count=1,\n",
        ")\n",
        "\n",
        "result = completion.result\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Ooyb6VOmdIf"
      },
      "source": [
        "The `stop_sequence` is not included in the result. Split off the expression and run it through the calculator, and add it back to the result:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K9Y0Z-Lome0P"
      },
      "outputs": [],
      "source": [
        "# Use re to clear units from the calculator expressions\n",
        "import re\n",
        "# Use numexpr since `eval` is unsafe.\n",
        "import numexpr\n",
        "\n",
        "\n",
        "def calculator(result):\n",
        "  result, expression = result.rsplit('<calc>', 1)\n",
        "\n",
        "  # Strip any units like \"cats / house\"\n",
        "  clean_expression = re.sub(\"[a-zA-Z]([ /a-zA-Z]*[a-zA-Z])?\",'', expression)\n",
        "\n",
        "  # `eval` is unsafe use numexpr\n",
        "  result = f\"{result}<calc>{expression}</calc> = {str(numexpr.evaluate(clean_expression))}\"\n",
        "  return result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bSAkZY3B2l9T"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "In each house, there are <calc>31 * 14</calc> = 434\n"
          ]
        }
      ],
      "source": [
        "print(calculator(result))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSwjboF3pz8E"
      },
      "source": [
        "Now append that to the prompt, and run the model again, so it can continue where it left off:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p1Q1rQ1E0XSB"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "mittens.\n",
            "In each house, there are <calc>31 * 6\n"
          ]
        }
      ],
      "source": [
        "continue_prompt=calc_prompt +\"\\n\"+ \"-\"*80 + \"\\n\" + calculator(result)\n",
        "\n",
        "completion = generate_text(\n",
        "    model=model,\n",
        "    prompt=continue_prompt,\n",
        "    stop_sequences=[\"</calc>\"],\n",
        "    # The maximum length of the response\n",
        "    max_output_tokens=800,\n",
        "    candidate_count=1,\n",
        ")\n",
        "\n",
        "print(completion.result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l-AY4lKu2JB9"
      },
      "source": [
        "This time, the model continued the text from the last calculation and moved on to the next. Now run it in a loop to fully solve the word problem:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TyIGLC4D1fF1"
      },
      "outputs": [],
      "source": [
        "def solve(question=question):\n",
        "  results = []\n",
        "\n",
        "  for n in range(10):\n",
        "    prompt = calc_prompt_template.format(question=question)\n",
        "\n",
        "    prompt += \" \".join(results)\n",
        "\n",
        "    completion = generate_text(\n",
        "        model=model,\n",
        "        prompt=prompt,\n",
        "        stop_sequences=[\"</calc>\"],\n",
        "        # The maximum length of the response\n",
        "        max_output_tokens=800,\n",
        "    )\n",
        "\n",
        "    result = completion.result\n",
        "    if '<calc>' in result:\n",
        "      result = calculator(result)\n",
        "\n",
        "    results.append(result)\n",
        "    print('-'*40)\n",
        "    print(result)\n",
        "    if str(answer) in result:\n",
        "      break\n",
        "    if \"<calc>\" not in  result:\n",
        "      break\n",
        "\n",
        "  is_good = any(str(answer) in r for r in results)\n",
        "\n",
        "  print(\"*\"*100)\n",
        "  if is_good:\n",
        "    print(\"Success!\")\n",
        "  else:\n",
        "    print(\"Failure!\")\n",
        "  print(\"*\"*100)\n",
        "\n",
        "  return is_good"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SBhW2JjF2WbJ"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "----------------------------------------\n",
            "The total number of cats is <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "The total number of mittens is <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "The total amount of yarn needed for the mittens is <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m.\n",
            "The total number of hats is <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            " The total amount of yarn needed for the hats is <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m.\n",
            "In total, <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n"
          ]
        }
      ],
      "source": [
        "solve(question);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b13884c9baeb"
      },
      "source": [
        "You can run that a few times to estimate the solve rate:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k5xyLTZbMjuF"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "They need <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "The mittens need <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            "They need <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            "The hats need <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            " They need a total of <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "So for the mittens, we need <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "That means we need <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn for mittens.\n",
            "For the hats, we need <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            "That means we need <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn for hats.\n",
            " In total we need <calc>787710 + 4711938</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "In the 77 houses I have <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "They need <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "The mittens need <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            "They need <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            "The hats need <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            " So, in total I need <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "The number of cats is <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            ". Each cat needs <calc>14 * 141</calc> = 1974\n",
            "----------------------------------------\n",
            "m of yarn for mittens. So we need <calc>1974 * 2387</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn for mittens. Each cat needs <calc>6 * 55</calc> = 330\n",
            "----------------------------------------\n",
            "m of yarn for hats. So we need <calc>330 * 2387</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn for hats. So in total we need <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "Each cat needs <calc>14 * 141</calc> = 1974\n",
            "----------------------------------------\n",
            "yarn for mittens.\n",
            "All cats need <calc>2387 * 1974</calc> = 4711938\n",
            "----------------------------------------\n",
            "yarn for mittens.\n",
            "Each cat needs <calc>6 * 55</calc> = 330\n",
            "----------------------------------------\n",
            "yarn for hats.\n",
            "All cats need <calc>2387 * 330</calc> = 787710\n",
            "----------------------------------------\n",
            "yarn for hats.\n",
            " All in all, you need <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "Each cat needs <calc>14 + 6</calc> = 20\n",
            "----------------------------------------\n",
            "items.\n",
            "So we need <calc>20 * 2387</calc> = 47740\n",
            "----------------------------------------\n",
            "items in total.\n",
            "Each mitten needs <calc>141</calc> = 141\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            "So all the mittens need <calc>141 * 47740</calc> = 6731340\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            " Each hat needs <calc>55</calc> = 55\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            "So all the hats need <calc>55 * 47740</calc> = 2625700\n",
            "----------------------------------------\n",
            "m of yarn.\n",
            " In total, we need <calc>6731340 + 2625700</calc> = 9357040\n",
            "----------------------------------------\n",
            "m of yarn. There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "Each cat needs <calc>14 + 6</calc> = 20\n",
            "********************************************************************************\n",
            "Failure!\n",
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "There are <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "There are <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            "There was <calc>141 * 33418</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn needed for mittens.\n",
            "There was <calc>55 * 14322</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn needed for hats.\n",
            " So there was <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "There are <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats in total. \n",
            "They need <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens. \n",
            "That's <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "meters of yarn for mittens. \n",
            "They need <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats. \n",
            "That's <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "meters of yarn for hats. \n",
            "So, they need <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "There are 77 houses * 31 cats / house = <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats.\n",
            "So we need <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens.\n",
            "So we need <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn for mittens.\n",
            "So we need <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats.\n",
            " So we need <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn for hats.\n",
            "In total, we need <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n",
            "----------------------------------------\n",
            "In total there are 77 houses * 31 cats / house = <calc>77 * 31</calc> = 2387\n",
            "----------------------------------------\n",
            "cats. In total 2387 cats * 14 mittens / cat = <calc>2387 * 14</calc> = 33418\n",
            "----------------------------------------\n",
            "mittens. In total 33418 mittens * 141m / mitten = <calc>33418 * 141</calc> = 4711938\n",
            "----------------------------------------\n",
            "m of yarn for mittens. In total 2387 cats * 6 hats / cat = <calc>2387 * 6</calc> = 14322\n",
            "----------------------------------------\n",
            "hats. In total 14322 hats * 55m / hat = <calc>14322 * 55</calc> = 787710\n",
            "----------------------------------------\n",
            "m of yarn for hats. In total we need 4711938 m of yarn for mittens + 787710 m of yarn for hats = <calc>4711938 + 787710</calc> = 5499648\n",
            "********************************************************************************\n",
            "Success!\n"
          ]
        }
      ],
      "source": [
        "good = []\n",
        "\n",
        "for n in range(10):\n",
        "  good.append(solve(question))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GoaPvoy-MjuN"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.9"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "np.mean(good)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "text_calculator.ipynb",
      "toc_visible": true
    },
    "google": {
      "image_path": "/static/site-assets/images/icon-palm.png",
      "keywords": [
        "examples",
        "palm",
        "samplecode",
        "python",
        "text"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
